Papers with compatibility function
Tensorized Self-Attention: Efficiently Modeling Pairwise and Global Dependencies Together (N19-1)
Copied to clipboard
| Challenge: | Neural networks equipped with self-attention have parallelizable computation and the ability to capture both long-range and local dependencies. |
| Approach: | They propose a novel attention mechanism called "Multi-mask Tensorized Self-Attention" it captures pairwise and global dependencies by a compatibility function composed of dot-product and additive attentions . |
| Outcome: | The proposed model outperforms CNN-/RNN-/attention-based models on nine NLP benchmarks with compelling memory- and time-efficiency. |